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Abstract 


The nature and effects of PBLA were investigated. I examined LINC program 
evaluations, government-solicited assessment reports, PBLA research, and 
other PBLA-related documents. I discuss the features of PBLA and its reported 
effects on language outcomes and teacher and student attitudes. I found that 
the government did not provide a rationale for PBLA and that the results of 
research did not support the introduction of PBLA. I also found that PBLA is 
neither standardized nor portfolio-based as claimed. It is costlier, more time- 
consuming, and appears to have more teacher pushback than the approach it 
replaced. Regardless, there is no evidence that the LINC program has improved 
students’ language skills before or after the implementation of PBLA 
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In 2010, Citizenship and Immigration Canada (CIC) introduced Portfolio-Based Language 
Assessment (PBLA) to improve the Language Instruction for Newcomers to Canada 
program (LINC) (CIC, 2010). With more than $200 million (FY 2015/16) in funding, 
it is the largest language program in Canada and it continues to grow (Immigration, 
Refugees and Citizenship Canada [IRCC], 2017). LINC is important to students because 
the achievement of certain LINC benchmarks can improve high-stakes citizenship and 
employment outcomes (CIC, 2010). 


PBLA was a large national intervention intended to deliver a new approach to language 
instruction and assessment to more than fifty thousand students (CIC, 2010). What was 
the rationale underpinning PBLA? What characterizes PBLA? What was its impact? To 
answer these questions, I examined PBLA research, program evaluations, reports, and 
other PBLA related documents (CIC, 2010; Fox, 2014; Fox & Fraser, 2012; Holmes, 2015; 
Immigration, Refugees and Citizenship Canada [IRCC], 2017; Pettis, 2014). I will begin by 
discussing LINC assessment reports and the PBLA pilot study. I will then look at claims 
that PBLA is standardized, authentic, collaborative, and portfolio-based. I will finish by 
describing PBLA’s impact on teachers’ attitudes, program costs, and student proficiency. 


Concepts of formative, summative, and standardized assessment vary in PBLA literature, 
so I will define these terms for clarity. Summative and formative assessments have different 
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purposes. Both assess current performance in relation to the outcomes, but formative 
assessments are ad-hoc, unstandardized, and used to ‘form’ or guide instruction and 
learning. A summative assessment, however, is planned and often standardized to some 
extent but has no formative role because it marks the end of an instructional unit. Scores 
on summative assessments are permanently recorded and represent value to stakeholders 
after the course is finished. Formative assessment scores are not recorded and have no 
value once the target outcome has been achieved. 


Standardized assessments have been characterized as externally developed, large-scale, 
and norm-referenced (Henning, 1987; Pettis, 2014). However, the defining feature of a 
standardized test is its reliability rather than its size or the type of inferences drawn from 
its scores. A standardized test requires “all test takers to answer the same questions, or 
a selection of questions from a common bank of questions, in the same way, and ... is 
scored in a ‘standard’ or consistent manner” (Abbot, 2014). A standardized assessment 
can be large- or small-scale, and inferences drawn from its scores can be either norm- or 


criterion-referenced. 


The Introduction of PBLA 


Pre-PBLA 


Just prior to introducing PBLA, CIC completed a comprehensive LINC program evaluation, 
which included the results of a survey and quasi-experimental study (CIC, 2010). In the 
program evaluation, CIC reported that the existing LINC program was cost-effective and 
that the teachers, materials, curriculum, and assessments were superior (pp. 14, 41, 42). 
The results of the survey and quasi-experimental study indicate otherwise. When surveyed, 
teachers reported that determining levels was confusing, that there were no progress and 
exit tests, that the program’s objectives were unclear, and that there was no standard 
curriculum (p. 62). In the quasi-experimental study, CIC (2010) concluded that there were 
no significant differences in the “four skill[s]” between those who had taken LINC classes 
and those who had not (pp. 34, 35). The $100 million (FY 2008/09) LINC program in 
place when PBLA was introduced was neither pedagogically effective nor cost effective as 
claimed earlier in the same document (CIC, 2010). 


Rationale for PBLA 


It is unclear why the government chose PBLA. Prior to the PBLA pilot study, external 
experts hired by the government to evaluate assessment models did not support 
assessments like PBLA (Makosky, 2008; Nagy & Stewart, 2009). In a literature review of 
alternative assessments Fox (2008), the same researcher who later conducted the pilot 
study, reported on the lack of reliability, high costs, and high labour demands that were 
associated with portfolios. These drawbacks had also been discussed by Nagy and Stewart 
in their report to CIC on LINC assessment (2009) prior to PBLA. The government-hired 
experts, including a pilot study researcher, had clearly described inherent problems with 
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PBLA-like assessments (Fox, 2008; Nagy & Stewart, 2009), yet there was no evidence that 
these particularly relevant issues and documents were ever considered, which suggests 
that the decision to implement PBLA in 2010 (CIC) was foregone. 


Four years after PBLA was introduced, experts (Pettis, 2014; Holmes, 2015) claimed that 
the decision to implement PBLA was based on the recommendations of other experts 
Makosky (2008) and Nagy and Stewart (2009). As mentioned above, these three experts 
did not support the use of portfolios as standardized assessments. Makosky did not even 
discuss the use of portfolios other than in a parenthetical comment about them being 
formative and controversial tools used by some school boards. Nagy and Stewart (2009) 
specifically warned against using portfolios as standardized assessments because of their 
low reliability, lack of validity, and high costs — problems acknowledged by the pilot study 
researcher prior to PBLA (Fox, 2008) and evident in the researcher’s subsequent PBLA 
studies (Fox & Fraser, 2012; Fox, 2014). 


The Pilot Study 


CIC’s first step in implementing PBLA was to pilot a “standardized portfolio-based 
assessment” that would “produce reports on student progress and the immediate outcomes 
of language training” (2010, p. x) to improve achievement and its measurement. The pilot 
study, however, focused on student and teacher behaviours and perceptions rather than 
achievement (Fox & Fraser, 2012). More than half of all the teachers surveyed reported 
that PBLA had no effect or a negative effect on their planning, teaching, and assessment 
(Fox & Fraser, 2012, p. 11). 


Likewise, there was little or no evidence that the implementation of PBLA improved 
students’ perceptions of how often they reviewed their work (Fox & Fraser, 2012, p. 20), 
their attendance (p. 23), their own fluency (p. 26), the effectiveness of the class materials 
(p. 26), and the organization of their work (p. 27). Insufficient rater-reliability, low teacher 
participation, and teacher complaints about PBLA were also reported. Teacher and student 
opposition to PBLA is convincing evidence that PBLA was ineffective, unpopular, and time- 
consuming, yet it was deemed a success by the government, the pilot study researchers, 
and an expert (CIC, 2013; Fox & Fraser; 2012, Pettis; 2012). 


The Nature of PBLA 


Formative and Summative Roles 


At the time it was introduced, the government and experts established PBLA as a formative, 
summative, and standardized assessment (CIC, 2010; Fox & Fraser, 2012; Pettis, 2012). 
Two years later, however, one of the experts, the pilot study researcher, stated that she 
and others had “got it wrong”, arguing that PBLA was not working as intended because 
it was used as a summative rather than a formative tool (Fox, 2014, p. 68). Other experts 
made similar arguments. Citing Black and Wiliam (1998a), Pettis (2012, 2014) and Holmes 
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(2015) claimed that all assessment must be formative, but Holmes also stressed the 
importance of good summative assessments. Teachers and other PBLA experts stated that 
summative testing would be unnecessary once PBLA was implemented (Elsa Net, 2015). 
These claims are inconsistent with the purpose of PBLA and the definitions of formative 
and summative assessments. Unlike formative assessments, summative assessments are 
necessary and their scores represent value outside the classroom. Furthermore, Black 
and Wiliam did not argue that all assessment should be formative, but they did state that 
summative assessments are required (1998b). 


PBLA as a Portfolio-based assessment 


Whether used formatively or summatively, PBLA lacks the characteristics of a true 
portfolio-based assessment. The PBLA “portfolio” is organized and scored according 
to the outcomes (the CLBs) but scores are not assigned to the portfolio itself. In a true 
portfolio-based system, the student, not the teacher, selects artefacts for assessment. 
Ultimately, the PBLA portfolio serves the same purpose as a typical binder — to improve 
the summative assessment outcome by organizing prescribed course work, including 


formative assessments, for review and study. 


PBLA as systematic, authentic, collaborative, and learner-centred 


PBLA has been defined as “...systematic, authentic, and collaborative” and learner-centred 
(Pettis, 2014, p.7; 2012, p. 18). PBLA is based on standardized outcomes but its assessment 
tasks are unstandardized because they are individually created or chosen (Pettis, 2014) by 
each teacher. In these respects, PBLA resembles a typical course with multiple sections and 
a shared exam. These teacher-chosen assessment tasks are authentic for the teacher but not 
the student. Furthermore, the involvement of individual teachers for artefact selection and 
the use of students’ peers for summative assessment tasks (Pettis, 2014) is unsystematic 
and raises bias and authorship issues (Gearhart & Herman, 1998). 


PBLA has been called collaborative and learner-centred (Pettis, 2012, 2014) but compared 
to students in a true portfolio or traditional system, PBLA students have less control. 
They have no role in the selection of artefacts for assessment (Ripley, 2012) nor do they 
want one (Fox, 2014). According to the PBLA guide (Pettis, 2014), teachers must choose 
artifacts because students choices were “unhelpful” (p.38). PBLA also prescribes much of 
the content and the use of student binders (Hajer, n.d.). Each binder must contain personal 
information, all class work for an entire benchmark, formative and summative assessment 


tasks for an entire benchmark, and 128 pages of reference material. 


Much of the reference material, which includes information on the House of Commons, 
how to buy a house, and the provincial and federal income tax brackets, is widely available 
online, rarely accessed, and not tested. Therefore, it is not surprising that students and 
teachers reported that the PBLA binder was unnecessary, cumbersome, and stressful 
to use and repeatedly carry home and to class (ELSA Net, 2015; Fox, 2014). Students in 
traditional and true portfolio systems have more control over what goes in their binder or 
portfolio and how it is organized. 
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The Effects of PBLA 


Effect on proficiency 


The goal of the LINC program isto help students settle in Canada by improving their language 
skills (IRCC, 2017), but the implementation of PBLA has had a negative overall effect. In 
the pilot study, students reported that PBLA did not improve their English (Fox & Fraser, 
2012). PBLA students also needed 50% more instructional time to complete a benchmark, 
and they used English outside of the home 20% less frequently than newcomers not in 
the program (IRCC, 2017; 2018). Most importantly, students who had been in the LINC 
program, pre- or post-PBLA, had not improved their English any more than newcomers 
not in the program (CIC, 2010, p. 35; IRCC, 2018, p. 9). Other negative effects associated 
with PBLA were high teacher pushback, higher labour requirements, and higher costs. 


Teacher Pushback 


Teacher complaints remain a prominent feature of PBLA seven years after it was introduced 
in 2010. Teacher pushback was clearly evident in early research (Fox, 2014; Ripley, 
2012) and more recently in a petition signed by a group of more than 650 people who are 
appealing to the minister responsible for the IRCC to abolish PBLA (Lachini, 2018). Based 
on the petition statement and the comments, it appears many teachers are unhappy with 
PBLA for similar reasons given by the teachers surveyed six years earlier in the pilot study 
(Fox, 2012). Teachers have opposed PBLA since its introduction but the current opposition 
to it is striking because it is widespread and strong. 


Labour Requirements 


Another important problem, albeit not directly pedagogical, is the number of teachers and 
instructional hours PBLA requires. The use of portfolios increases labour requirements 
and costs, which can be significant (Catteral & Winters, 1994; Hargreaves, Earl, & Schmidt; 
2002; Nagy & Stewart, 2009), and PBLA is no exception. Language instruction costs, which 
were highly correlated with the number of teachers (CIC, 2010), had changed very little in 
the five years prior to PBLA. However, four years after PBLA was introduced, assessment 
costs rose 220% while the number of students rose only 75% (CIC, 2010; IRCC, 2017). 
Teacher complaints about the additional time demands and unpaid work under PBLA 
(Fox, 2014; Lachini, 2018; Ripley, 2012) suggest that the large increase in funding was 


insufficient and that its labour requirements are still unmet. 


Conclusions 


First, the decision to implement PBLA was neither evidence-based nor clear. The results 
of the pilot study and the two government reports on assessment (Fox & Fraser, 2012; 
Makosky, 2008; Nagy and Stewart, 2009) did not support the implementation of a system 
like PBLA. Second, conceptions of portfolios and standardized, summative, and formative 
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assessments were inconsistent with their definitions and the purpose of PBLA. Third, PBLA 
is neither portfolio-based nor standardized. It is an assessment based on shared outcomes, 
which are standards, but it uses unstandardized tasks. Fourth, teachers and students view 
PBLA as onerous and ineffective. Fifth, PBLA and the previous approach were equally 
ineffective in changing language outcomes in Canada’s $200 million (FY 2016) national 
language program (IRCC, 2017). PBLA, however, was costlier and required many more 


instructional hours. 
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